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Promising Evidence of ABCmouse Closing 
the Achievement Gap and Minimizing 


summer Learning Loss: 


A Large-Scale Study of Children Ages 2-8 


Key Findings 

e Based on a sample of more than 14,000 
children ages 2-8, using ABCmouse for at 
least 45 minutes and completing 15 Learning 
Activities across three separate days per week 
can increase learning and minimize summer 
learning loss. 


e Children who use ABCmouse more— 
whether by completing structured lessons or 
unstructured free play activities—show greater 
learning gains. 


e Regular ABCmouse usage can help children of 
all skill levels, particularly those who start the 
summer with lower skills, catch up to higher- 
skilled peers by summer’s end, closing the gap 
by up to 87 percent. 


e More than 80 percent of parents reported 
that, since using ABCmouse, their children 
were more motivated to learn and were more 
independent in completing Learning Activities. 


Overview 


During the summer months, school-based learning is 
typically no longer a part of children’s daily lives. While 
children in middle- and higher-income families may 
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continue to learn during the summer from educational 
resources and enrichment activities their families can 
provide, children in less-affluent families are more 
likely to lose access to books, educators, meals, and 
organized activities when schools close.' Achievement 
gaps between students of different socioeconomic 
groups widen more during the summer than during the 
school year (i.e., summer learning loss), largely due to 
unequal learning opportunities in children’s homes and 
communities.2 Exacerbating this phenomenon, COVID- 
19-related school closures have kept tens of millions of 
children away from schools for many months, which has 
been particularly detrimental to children in low-income 
families.? 


Schmidt Futures, a philanthropic organization that 
supports educational and commercial efforts with a 
public purpose, awarded a grant to WestEd to conduct 
an exploratory study focused on how parents used 
ABCmouse to support learning at home during the 
summer of 2020. WestEd is a nonprofit research, 
development, and service agency with deep expertise in 
evaluating the impact of educational programs. For this 
first independently funded study of ABCmouse, WestEd 
researchers collaborated with Age of Learning’s in- 
house Efficacy Research team on study design and data 
analysis. Age of Learning recruited the participants and 
oversaw data collection. 


Alexander, K.L., Entwisle, D.R., & Olson, L. S. (2007). Lasting consequences of the summer learning gap. American Sociological Review, 72 (2), 167-180. 
Pitcock, S. (2018). The case for summer learning: why supporting students and families all year is vitally important. American Educator, 42 (1), 4-8. 
Dorn, E., Hancock, B., Sarakatsannis, J. & Vriuleg, E. (2020). COVID-19 and learning loss—disparities grow and students need help. McKinsey & Company. 
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Program 


ABCmouse Early Learning Academy is a comprehensive 
digital early learning resource for children in preschool, 
prekindergarten, kindergarten, 1st, and 2nd grades. 
Informed by research on how young children learn 

and what motivates them to learn, ABCmouse offers 
more than 10,000 Learning Activities and 850 lessons 
encompassing all major academic subject areas, 
including literacy, math, science, social studies, health, 
art, and music. ABCmouse Learning Activities are 
regularly updated to encourage children to explore new 
content based on studies demonstrating connections 
between curiosity, exploration, and learning.* 
Additionally, given the research showing relationships 
between learner choice and learning, ABCmouse offers 
games, books, puzzles, songs, and videos that children 
can choose independently based on their interests, 

as well as lessons on the Step-by-Step Learning Path, 

a structured curriculum that provides predesignated 
activities at the child’s level.” In sum, ABCmouse 
focuses on making learning fun, fostering children’s 
curiosity, and promoting their confidence as learners, 
all of which can contribute to children’s cognitive 

and socioemotional development. It can be used on 
smartphones and tablets as an app, as well as on 
computers through a web browser. In 2020, more than 
10 million children used ABCmouse. 


Participants 


In March 2020, Age of Learning launched its School 
Continuity Initiative (SCI) to make its digital learning 
programs available for free to teachers and families 
affected by school closures. Through this initiative, more 


than 825,000 new accounts were created, providing 
more than 1 million children access to ABCmouse at 
home. In June 2020, all active users of ABCmouse, 
including paying subscribers who had completed at 
least 15 activities in the prior five days, received an email 
invitation to complete a parent survey and have their 
children complete a pre-assessment and then a post- 
assessment in late August/early September. Another 
email was sent in late August to then-active users, 
whether or not they had completed the pre-assessment, 
asking them to complete the parent survey and have 
their children complete the post-assessment. A total of 
6,044 children completed the pre-assessment, 

and 14,489 completed the post-assessment 

(ages 2—4.5 = 6,936; ages 4.6-6.5 = 10,869; 

ages 6.6-9 = 2,581; mean age = 5.19, SD = 1.84). 
Additionally, 5,990 individuals across the U.S. 
completed the parent survey (60.5% White/Caucasian; 
13.5% Latinx; 10.3% Asian/Asian-American; 6.9% Black/ 
African-American). Approximately 30 percent of the 
analytic sample had gained access to ABCmouse 
through the SCI. 


Design and Procedures 


We used the ABCmouse Assessment Center, a 
comprehensive suite of standards-aligned, validated 
digital assessments that measure a wide range of literacy 
and math skills across preschool, prekindergarten, 
kindergarten, 1st, and 2nd grades. The Assessment 
Center includes thousands of online assessment 

items developed in collaboration with experts at SRI 
International and NORC at the University of Chicago, and 
can be used to evaluate a child’s learning progress. 
Table 1 displays the assessed skills. 


4 Wolfe, P. (2006). The role of meaning and emotion in learning. New Directions for Adult and Continuing Education, 110, 35—41. 


5 Hirsh-Pasek, K. & Golinkoff, R. M. (2008). Why play = learning. In Tremblay, R.E., Boivin, M., Peters, RDeV, (Eds.) Encyclopedia on Early Childhood 
Development [online], 1—6. Montreal, Quebec: Centre of Excellence for Early Childhood Development and Strategic Knowledge Cluster on Early Child 


Development 
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Table 1. Skills Across Preschool, Prekindergarten, Kindergarten, 1st, and 2nd Grades Assessed Using the ABCmouse Assessments 


ee ee ee 


e Picture vocabulary 

- Upper- and lowercase letter identification 
e Rhyming words 

e Beginning and consonant sounds 
e Replacing sounds in words 

e Verb tenses 

e Root words 

e Singular and plural nouns 

e Subject-verb agreement 

e Prefixes and suffixes 

e Adjectives and adverbs 

e Reading comprehension 


Children saw a pop-up window in ABCmouse prompting 
them to take the appropriate grade-level assessment. 
Families who completed both the pre- and post- 
assessments as well as the parent survey received three 
free months of ABCmouse access. 


Among the 6,044 children who completed the pretest 
and the 14,489 children who completed the posttest, 
only 349 completed both. Using the data from 20,533 
children who completed either a pretest or a posttest, 
we determined that the reliability coefficients associated 
with the pretests were .88, .78, .92, .96, .91 across 

the grades and .89, .94, .95, .91, .93 for the posttests. 
Then, the data from all 20,533 children who completed 
either a pretest or a posttest were used in multiple 


e Recognizing numerals 1-25 

e Counting objects 1-20 

e Comparing quantities and sizes 
e Recognizing 2D shapes 

e Adding numbers up to 100 

e Subtracting numbers up to 100 
e Comparing numbers 

e Identifying place values 

e Reading data and graphs 

e Partitioning and fractions 

e Estimating length 


imputation, then deletion (MID), a technique for handling 
missing data, to estimate the pretest scores of 14,489 
children who completed only the post-assessment.® 
The MID method took into account all 20,533 children’s 
demographic characteristics (age, gender, grade); 
ABCmouse usage (total time spent, total activities 
completed, total lessons completed, unique days); and 
all available pretest and posttest scores to estimate 

the missing pretest scores of 14,140 children. The final 
analytic sample was the 14,489 children for whom we 
had posttest data and imputed pretest data. 


On average, children used ABCmouse for 53.53 minutes 
per week (SD = 44.93) and completed 11.75 (SD = 9.49) 
Learning Activities per week over 10—11 summer weeks. 


6 Von Hippel, P.T. (2007). Regression with missing Ys: an improved strategy for analyzing multiply imputed data. Sociological Methodology. 37 (1), 83—117. 


doi:10.1111/j.1467-9531.2007.00180.x 
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Results 


Finding 1. Using ABCmouse for at least 45 minutes and completing 15 Learning Activities across at least three 
separate days per week can increase learning and minimize summer learning loss. 


Examining the relationship between ABCmouse usage and performance, the metrics most significantly related to 
overall posttest scores (literacy and math combined) were average days of gameplay per week, average activities 
completed per week, and average time spent per week. Figures 1, 2, and 3 show that using ABCmouse for at least 
45 minutes and completing at least 15 activities across at least three unique days per week is associated with greater 
improvements in key math and literacy skills. In other words, more time spent engaging in ABCmouse is related to 
higher posttest performance. 


os Average Time Spent Per Week (minutes) 
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Figure 1. Spending at least 45 minutes on ABCmouse per week is associated with higher posttest performance. 
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Figure 2. Completing at least 15 Learning Activities on ABCmouse per week is associated with higher posttest performance. 
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Figure 3. Using ABCmouse for three or more days on average per week is associated with higher posttest performance. 
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Finding 2. The more children use ABCmouse—whether by completing structured lessons or free play activities— 
the more they learn. 


To better understand children’s different learning experiences using ABCmouse, we conducted a latent profile analysis. 
This was done to identify different groups based on levels of usage as well as completion of structured lessons on 

the Step-by-Step Learning Path (structured learning) versus completion of Learning Activities in free play areas, where 
children can choose activities of different types in any order (unstructured learning). The resulting best-fitting model 
produced four groups, shown in Figure 4. The median weekly usage for each group is displayed in Table 2. 
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Figure 4. Four groups of children can be distinguished by the amount of usage and by whether they completed lessons or free play activities 
on ABCmouse. 


Table 2. Median Weekly Usage of ABCmouse for Each Group 


Median Time 
Median Activities EREDETE 
Spent Weekly 
Completed Weekly Per Week 


(minutes) 





Low Engagement AAAS! 5 0.38 
Moderate Engagement 60.64 13 1.24 
High Engagement Structured Learning 95.64 26 1.52 
Very High Engagement 13608 25 2 AMO 


Unstructured Learning 
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Comparing the posttest performance of the four user groups (Figure 5) indicates that greater usage of ABCmouse, 
whether by completing lessons or free play activities, was associated with a positive impact on posttest performance. 
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(Group 3, N=1,017 or 7.02%) (Group 4, N=1,818 or 12.55%) 


Figure 5. Comparisons of group 1 with the remaining groups show that the more children use ABCmouse, the more they learn, regardless of 
the type of Learning Activities they complete. 


Finding 3. Regular ABCmouse usage can help children of all skill levels, particularly those who start the summer 
with low skill levels, catch up to their higher-skilled peers by summer’s end, closing the gap by up to 87 percent. 


We next compared children in the low-engagement group with those in the very high-engagement group. For each 
child in the very high-engagement group, we identified a counterpart in the low-engagement group who shared similar 
pretest scores, age, and identical gender, grade, and pretest and posttest form. Then, we weighted the children in 
each group to establish similarly sized groups. For example, if there was one very high-engaged child matched to 
three low-engaged children, the very high-engaged child was weighted more heavily to establish a balance between 
the two groups being compared. The goal was to create two similarly sized groups matched on all observable 
characteristics except for level of ABCmouse usage. 


A weighted regression analysis comparing the low-engagement and very high-engagement groups (Groups 1 and 
4) after controlling for age, gender, and pretest performance, showed that the effect of very high engagement with 
ABCmousée relative to low engagement is related to an increase of 4.35 percent on the posttest. This statistically 
significant result represents an effect size of 0.2 (Figure 6). 
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Figure 6. The effect of greater ABCmouse usage during the summer is a statistically significant 4.35 percent boost on the posttest. 
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We then compared the pretest and posttest performance of the low-engaged and very high-engaged children across 
initial performance on the pretest, as shown in Figure 7. Regular, consistent use of ABCmouse benefited all children, 
regardless of their skill level at the start of the summer. 


70 
68 


66 


64 


Posttest Score 


62 


60 





58 


0-25th percentile 26-50th percentile 51-75th percentile 76-100th percentile 
Pretest Performance 


E Low Engagement High Engagement 


Figure 7. Regular ABCmouse usage helps children across all skill levels. 


Among children with high prior knowledge, posttest scores increased from 64.2 to 67.9 percent, representing 

an increase rate of 5.8 percent [(67.92-64.18)/6418 = .058]. Among children with low prior knowledge, posttest 
scores increased from 59.2 to 63.5 percent, which represents a higher increase rate of 7.2 percent [(63.52- 
59.23)/59.23 = .072]. In particular, with greater, consistent ABCmouse usage, children who started the summer with 
low prior knowledge caught up with their peers with high prior knowledge (but with low engagement), closing the 
performance gap between the groups by up to 87 percent [(63.52-59.23/64.18-59.23) = .867], as shown in Figure 8. 
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Figure 8. Low engagement with ABCmouse is associated with summer learning loss, while high engagement is associated with learning gains. 
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Finding 4. More than 80 percent of parents reported that since using ABCmouse, their children were more 
motivated to learn and were more independent in completing Learning Activities. 


Of the 5,990 parents of who completed the survey, 84 percent indicated that since using ABCmouse, their children 


have become more independent in completing Learning Activities (Figure 9). Additionally, 82 percent indicated that 


their children were more motivated to learn about the topics they saw on ABCmouse. This is noteworthy, considering 
the challenges many children experienced in staying interested in schoolwork in the absence of the social interactions 


and structure provided by in-person schooling. 


| believe that ABCmouse has helped my child... 


be more independent in completing learning activities 

be more motivated to learn about what they saw in ABCmouse 
have more interest in learning more 

have more curiosity about things 

demonstrate mastery of content/skills covered in ABCmouse 

be more comfortable in trying new things 

apply what they learned in the program to other things in daily life 


be more persistent in finishing tasks even if they were somewhat difficult 


E Strongly Agree 
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Figure 9. A majority of parents indicated that ABCmouse had a positive impact on their children’s learning behaviors. 


Conclusion 


This study, the first independently funded efficacy 
evaluation of ABCmouse, included large numbers of 
children across all ages for whom the product was 
designed. The results showed that using ABCmouse 
consistently over 10—11 weeks—whether by completing 
structured lessons or unstructured free play activities— 
is associated with significant increases in summer 
learning, suggesting that ABCmouse is a promising 
resource to mitigate summer learning loss. ABCmouse 
usage benefited children of all skill levels. But the 
effects were especially noteworthy for those who 
started out with low skill levels and could catch up 


to their peers with higher skill levels through regular, 
consistent usage of ABCmouse, substantially closing 
the performance gap. Parents also valued ABCmouse’s 
engaging educational content that motivated their 
children to continue learning at home while becoming 
more comfortable and confident about using technology 
and developing greater independence and curiosity 

to explore new topics. In sum, this study highlights the 
educational value of ABCmouse as a program that 
helps children continue learning outside of school while 
enjoying the learning experience. 
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